feat(lens): estimate OTEL costs with shared Rust pricing by tin-berri · Pull Request #45111 · BerriAI/litellm

tin-berri · 2026-10-07T18:03:29Z

TLDR

Direct calls show tokens but leave Lens cost empty
Estimate supported costs using shared Rust pricing
Keep incomplete search, tool and routing charges unknown

Intentional product change: eligible direct calls display labeled estimates; unsupported charges remain unknown

User Flow

Before: a direct cached call has no visible cost

Send a recorded Moyai trace to POST /v1/traces
Open /lens and select the model call
See 3,010 input tokens and an empty cost

After: the same trace has a labeled estimate

Send the recorded Moyai trace to POST /v1/traces
Open /lens and select the model call
See Estimated $0.0013 and its pricing source

Relevant issues

Estimate direct-call costs from model, token and cache usage
Share the Rust cost crate and LiteLLM pricing catalog
Builds on gateway spend reconciliation from merged fix(lens): refresh runs until gateway costs are complete #45105

Changes

Select rates for recorded cache durations, reasoning, service tiers and call times
Preserve GenAI/OpenInference evidence across list, detail and batch queries
Set catalog supports_token_only_pricing for hosted search and routing entries
Resolve deployment paths and aliases to routing requirements independently of served token rates
Preserve Python gateway/SDK billing and recorded spend precedence, including zero

Screenshots / Proof of Fix

Local gateway, rebuilt standalone Rust Lens and real ClickHouse; labeled synthetic controls and previously captured provider usage

Local demo and actual HTTP recording · Screenshot · Receipt

Before (`d3e867a`)

Export azure_ai/model_router/my-deployment, served gpt-4o, with 100 input and 10 output tokens
Read GET http://127.0.0.1:4691/v1/traces/000000000000000000000000000e7201
The incomplete token-only total is $0.00035, omitting the routing fee

After (`8dbee5f`)

Read the same persisted spans through http://127.0.0.1:4691/v1/traces
Unsupported search, tool and routing charges remain unknown in list and detail
Ordinary text, optional search, zero usage and client tools retain estimates
Cached direct usage remains $0.00125758; recorded gateway spend remains $0.00002455 and $0

Validation

1,370 focused Rust tests and 98 Python catalog/schema tests pass
20 routed-identity regressions fail before the fix; all pass afterward
26 live controls agree between list and detail
Three cache-price comparisons agree with existing Python billing
Rust Clippy, formatting, make check and the unchanged base catalog validator pass
Greptile 5/5 on 8dbee5ff; Bugbot found no issues; Veria passed

CI status

Catalog guard, Python schema/metadata tests, code quality and Rust lint pass
Rust beta-registry failure also exists in the exact base sources
Push wheel retention assertion is already corrected on the current base
Vertex memory benchmark measured 0.62 against a 0.60 threshold; unchanged sources passed the previous run
Other CI jobs remain in progress; this PR is not all green

Pre-Submission checklist

Added meaningful tests for estimation and unknown costs
Focused tests and live verification pass
Current-head CI completed
Scope is Lens cost estimation
Current-head Greptile confidence is at least 4/5

Caveats

Medium

Estimates use bundled catalog prices, excluding private billing arrangements
Missing evidence or unsupported charges keep cost unknown

Low

Demo and recording are local artifacts
Other CI jobs are still running

tin-berri · 2026-10-07T18:09:08Z

@greptileai Please review this direct OTEL pricing change, including gateway precedence, cache token accounting, duplicate estimates, and list/detail integration.

codspeed · 2026-10-07T18:15:57Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_lens_otel_cost (8dbee5f) with main (406514f)}

greptile-apps · 2026-10-07T18:20:01Z

Confidence Score: 5/5

[High impact] The PR appears safe to merge, with both numbered findings addressed and no new blocking issue established

Summary

Lens adds labeled cost estimates for eligible direct calls using the shared Rust calculator and bundled model prices

Requested routing requirements remain attached when the served model differs
Unsupported charges and incomplete pricing evidence keep costs unknown
Recorded gateway spend, including zero, takes precedence
Named regression cases address both numbered findings

This review inspected code and tests; it did not run tests or live UI checks

_{Reviews (17) · Last reviewed commit: "feat(lens): estimate OTEL costs with sha..." · Reviewed by Greptile}

codecov · 2026-10-07T18:20:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

veria-ai · 2026-10-07T18:21:32Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

tin-berri · 2026-10-07T18:39:02Z

@greptileai Please review a8ca357, which addresses all previous findings and adds offline pricing, consistent call grouping, and batch regressions

tin-berri · 2026-10-07T18:39:04Z

bugbot run

Please review the current tip a8ca357 after the cost identity, batching, and offline pricing fixes

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-07T19:12:29Z

@greptileai Please review this final tip for tier-only pricing, strict selected-rate validation, and used cache-duration handling; all focused tests pass

tin-berri · 2026-10-07T19:12:31Z

bugbot run

Please review the final tip with tier-only pricing, strict selected-rate validation, and cache-duration regressions passing locally

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-07T19:51:02Z

@greptileai Please review 52ca0e2 for GenAI compatibility, captured call time, and reasoning pricing. All three findings have regression coverage

tin-berri · 2026-10-07T19:51:03Z

bugbot run

Please review 52ca0e2 for GenAI alias compatibility, stored timestamps, and reasoning pricing after the latest fixes and regression tests

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-08T03:09:13Z

@greptileai Please review 71d7e0a against gateway base e3d70ec, focusing on native pricing ownership, catalog parity, and late gateway reconciliation.

tin-berri · 2026-10-08T03:09:18Z

bugbot run

Please review 71d7e0a against gateway base e3d70ec, including standalone native estimation, bundled catalog assets, and authoritative gateway amounts

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-08T21:03:22Z

@greptileai Please review the missing search-charge fix, including canonical model admission, reported usage, ordinary text controls, and all four storage projections

tin-berri · 2026-10-08T21:03:24Z

bugbot run

Please recheck search cost admission and verify normal token estimates and recorded gateway spend precedence remain intact

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-08T21:39:52Z

@greptileai please review b64de19: catalog-owned pricing requirements replace model-name heuristics, preserving request context and declining incomplete tool charges

tin-berri · 2026-10-08T21:39:54Z

bugbot run

Please review b64de19 for catalog pricing requirements, requested-model context, and unsupported tool usage; gateway billing execution remains unchanged

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-08T21:49:21Z

@greptileai please review d3e867a: pricing eligibility uses catalog supports_token_only_pricing metadata, compatible with base validation; request context and tool-usage regressions pass

tin-berri · 2026-10-08T21:49:22Z

bugbot run

Please review d3e867a, including catalog schema compatibility, requested-model pricing requirements, and tool usage; Python gateway billing remains unchanged

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-08T22:06:39Z

@greptileai Please review 8dbee5f, including catalog routing contexts and named regression cases; gateway billing remains unchanged and live controls pass

tin-berri · 2026-10-08T22:06:41Z

bugbot run Please review 8dbee5f for routing context preservation, catalog admission, and regressions; gateway and SDK billing remain on Python

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8dbee5f. Configure here.}

greptile-apps Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread litellm-rust/crates/traces/src/resolve/estimates.rs Outdated

Comment thread litellm-rust/crates/traces/src/resolve/estimates.rs Outdated

Comment thread litellm-rust/crates/traces-cache/src/list.rs Outdated

Comment thread litellm/llms/base_llm/otel_cost.py Outdated

veria-ai Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread litellm/rust_bridge/trace/pricing.py Outdated

tin-berri force-pushed the litellm_lens_otel_cost branch from 4e916d2 to a8ca357 Compare October 7, 2026 18:38

cursor Bot reviewed Oct 7, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread litellm/llms/base_llm/otel_cost.py Outdated

tin-berri force-pushed the litellm_lens_otel_cost branch from a8ca357 to 8b441a4 Compare October 7, 2026 19:11

cursor Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread litellm/llms/base_llm/otel_cost.py Outdated

greptile-apps Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread litellm/llms/base_llm/otel_cost.py Outdated

Comment thread litellm/llms/base_llm/otel_cost.py Outdated

tin-berri force-pushed the litellm_lens_otel_cost branch from 8b441a4 to 52ca0e2 Compare October 7, 2026 19:50

cursor Bot reviewed Oct 7, 2026

View reviewed changes

tin-berri force-pushed the litellm_lens_otel_cost branch from 52ca0e2 to 71d7e0a Compare October 8, 2026 03:09

tin-berri requested a review from a team October 8, 2026 03:09

tin-berri changed the base branch from main to litellm_lens_gateway_cost October 8, 2026 03:09

tin-berri closed this Oct 8, 2026

tin-berri reopened this Oct 8, 2026

tin-berri had a problem deploying to e2e-changed October 8, 2026 03:11 — with GitHub Actions Error

cursor Bot reviewed Oct 8, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread litellm-rust/crates/traces/src/pricing.rs

tin-berri force-pushed the litellm_lens_otel_cost branch from 69a3cf7 to d61880b Compare October 8, 2026 21:02

tin-berri had a problem deploying to e2e-changed October 8, 2026 21:02 — with GitHub Actions Error

greptile-apps Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread litellm-rust/crates/traces/src/pricing.rs Outdated

cursor Bot reviewed Oct 8, 2026

View reviewed changes

tin-berri force-pushed the litellm_lens_otel_cost branch from d61880b to b64de19 Compare October 8, 2026 21:39

tin-berri requested review from kerry-berri, mateo-berri and ryan-crabbe-berri as code owners October 8, 2026 21:39

tin-berri had a problem deploying to e2e-changed October 8, 2026 21:39 — with GitHub Actions Error

cursor Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread litellm-rust/crates/traces/src/pricing.rs Outdated

greptile-apps Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread litellm-rust/crates/traces/src/pricing.rs Outdated

tin-berri force-pushed the litellm_lens_otel_cost branch from b64de19 to d3e867a Compare October 8, 2026 21:49

tin-berri had a problem deploying to e2e-changed October 8, 2026 21:49 — with GitHub Actions Error

cursor Bot reviewed Oct 8, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread litellm-rust/crates/traces/tests/pricing.rs Outdated

feat(lens): estimate OTEL costs with shared Rust pricing

8dbee5f

tin-berri force-pushed the litellm_lens_otel_cost branch from d3e867a to 8dbee5f Compare October 8, 2026 22:06

tin-berri requested a deployment to e2e-changed October 8, 2026 22:06 — with GitHub Actions Waiting

cursor Bot reviewed Oct 8, 2026

View reviewed changes

tin-berri closed this Oct 9, 2026

Conversation

tin-berri commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Changes

Screenshots / Proof of Fix

Before (d3e867a)

After (8dbee5f)

Validation

CI status

Pre-Submission checklist

Caveats

Medium

Low

Uh oh!

tin-berri commented Oct 7, 2026

Uh oh!

codspeed Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

greptile-apps Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

veria-ai Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Security review

Uh oh!

tin-berri commented Oct 7, 2026

Uh oh!

tin-berri commented Oct 7, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tin-berri commented Oct 7, 2026

Uh oh!

tin-berri commented Oct 7, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tin-berri commented Oct 7, 2026

Uh oh!

tin-berri commented Oct 7, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tin-berri commented Oct 8, 2026

Uh oh!

tin-berri commented Oct 8, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tin-berri commented Oct 7, 2026 •

edited

Loading

Before (`d3e867a`)

After (`8dbee5f`)

codspeed Bot commented Oct 7, 2026 •

edited

Loading

greptile-apps Bot commented Oct 7, 2026 •

edited

Loading

codecov Bot commented Oct 7, 2026 •

edited

Loading

veria-ai Bot commented Oct 7, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading